Skip to content

feat(operator): scaffold DGD placement score status wiring#11057

Open
ashnamehrotra wants to merge 10 commits into
ai-dynamo:mainfrom
ashnamehrotra:grove-placement-score
Open

feat(operator): scaffold DGD placement score status wiring#11057
ashnamehrotra wants to merge 10 commits into
ai-dynamo:mainfrom
ashnamehrotra:grove-placement-score

Conversation

@ashnamehrotra

@ashnamehrotra ashnamehrotra commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Part of DEP: Grove status observability in DynamoGraphDeployment #10064
  • Adds DGD-level placement score API fields:
    • status.placementScore
    • status.placementScoreState
  • Mirrors these fields across v1beta1 and v1alpha1, including conversion wiring and conversion field-set acknowledgment.
  • Wires placement score state update into DGD reconcile on the Grove pathway.
  • Adds placement score aggregation scaffolding in Grove integration:
    • Returns explicit non-failing states for current upstream limitations.
    • Leaves placementScore unset until Grove exposes a score field in a consumable API shape.
  • Adds focused tests:
    • DGD status conversion round-trip includes placement fields.
    • Grove aggregation behavior is pinned for current states (Unsupported / NotReported).

Validation

  • GOCACHE=/tmp/dynamo-go-cache go test ./api/v1alpha1 ./api/v1beta1 -count=1
  • GOCACHE=/tmp/dynamo-go-cache go test ./api/v1alpha1 ./internal/dynamo -count=1
  • GOCACHE=/tmp/dynamo-go-cache go build ./... (from deploy/operator)

Scope Notes

  • This PR intentionally does not implement low-score metrics/events yet.
  • This PR intentionally does not emit Reported / Partial yet, because current Grove API does not expose placementScore where Dynamo can read it.
  • Ready condition diagnosis behavior is unchanged.

Open in Devin Review

Summary by CodeRabbit

  • New Features
    • Added placement score reporting to deployment status, including a state that indicates whether the score is reported, partial, unsupported, not reported, or unknown.
    • Deployment reconciliation now surfaces placement score information when available for the Grove path.
  • Bug Fixes
    • Preserved placement score data during status conversions and round-trips between versions.
    • Updated status copying so placement score values are retained correctly.

Signed-off-by: ashnamehrotra <ashnamehrotra@gmail.com>
@ashnamehrotra
ashnamehrotra requested a review from a team as a code owner June 29, 2026 18:20
@copy-pr-bot

copy-pr-bot Bot commented Jun 29, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@ashnamehrotra
ashnamehrotra temporarily deployed to external_collaborator June 29, 2026 18:20 — with GitHub Actions Inactive
@github-actions

github-actions Bot commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

@github-actions github-actions Bot added feat external-contribution Pull request is from an external contributor labels Jun 29, 2026
@dynamo-ops

Copy link
Copy Markdown
Contributor

/ok to test b29bb9a

@github-actions github-actions Bot added the deployment::k8s Relates to dynamo deployment in kubernetes label Jun 29, 2026
@datadog-official

This comment has been minimized.

@coderabbitai

coderabbitai Bot commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Walkthrough

Adds PlacementScore (*float64) and PlacementScoreState (new enum) fields to DynamoGraphDeploymentStatus in both v1beta1 and v1alpha1 APIs. A new AggregatePlacementScore helper computes these from a PodCliqueSet; the controller's Grove-pathway reconcile loop is wired to call it. Conversion, deepcopy, and tests are updated accordingly.

Changes

Placement Score for DynamoGraphDeployment

Layer / File(s) Summary
API type definitions (v1beta1 and v1alpha1)
deploy/operator/api/v1beta1/common.go, deploy/operator/api/v1beta1/dynamographdeployment_types.go, deploy/operator/api/v1beta1/zz_generated.deepcopy.go, deploy/operator/api/v1alpha1/dynamographdeployment_types.go, deploy/operator/api/v1alpha1/zz_generated.deepcopy.go
Introduces the PlacementScoreState string enum with five constants in both API versions and adds PlacementScore *float64 and PlacementScoreState fields to DynamoGraphDeploymentStatus, with generated deepcopy for the pointer field.
AggregatePlacementScore helper and tests
deploy/operator/internal/dynamo/grove.go, deploy/operator/internal/dynamo/grove_test.go
Adds AggregatePlacementScore(pcs *PodCliqueSet) returning (nil, Unsupported) for nil/missing score and (nil, NotReported) for empty pod gangs; table-driven tests cover all three current code paths.
Controller reconcile wiring
deploy/operator/internal/controller/dynamographdeployment_controller.go
On the Grove pathway in Reconcile, fetches the existing PodCliqueSet, calls AggregatePlacementScore, and writes results to dynamoDeployment.Status; fetch errors are logged non-fatally.
Conversion round-trip and coverage
deploy/operator/api/v1alpha1/dynamographdeployment_conversion.go, deploy/operator/api/v1alpha1/dynamographdeployment_conversion_test.go, deploy/operator/api/v1alpha1/conversion_field_coverage_test.go
Extends both directions of DynamoGraphDeploymentStatus conversion to preserve PlacementScore and PlacementScoreState; round-trip test and field-coverage acknowledgment set are updated to match.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (2 warnings)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 71.43% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
Description check ⚠️ Warning The description covers the change and validation, but it does not follow the required template or include the required Related Issues section. Reformat it to the repo template with Overview, Details, Where should reviewer start?, and Related Issues; add the required issue reference or no-related-issue confirmation.
✅ Passed checks (3 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title clearly summarizes the main change: adding DGD placement score status wiring.

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@deploy/operator/internal/controller/dynamographdeployment_controller.go`:
- Around line 292-300: Reset the placement status fields in
dynamographdeployment_controller.go so stale values do not survive non-Grove or
lookup-failure reconciles. In the reconcile logic around r.isGrovePathway and
getExistingGrovePodCliqueSet, explicitly clear or set
dynamoDeployment.Status.PlacementScore and
dynamoDeployment.Status.PlacementScoreState on the non-Grove branch and when
pcsErr is returned, instead of only updating them after AggregatePlacementScore
succeeds. Use the existing symbols r.isGrovePathway,
getExistingGrovePodCliqueSet, and dynamo.AggregatePlacementScore to place the
fix in the same status-update path.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 03ddbb02-5731-442a-8efc-7bfe4bb28257

📥 Commits

Reviewing files that changed from the base of the PR and between 47f19f2 and b29bb9a.

📒 Files selected for processing (11)
  • deploy/operator/api/v1alpha1/conversion_field_coverage_test.go
  • deploy/operator/api/v1alpha1/dynamographdeployment_conversion.go
  • deploy/operator/api/v1alpha1/dynamographdeployment_conversion_test.go
  • deploy/operator/api/v1alpha1/dynamographdeployment_types.go
  • deploy/operator/api/v1alpha1/zz_generated.deepcopy.go
  • deploy/operator/api/v1beta1/common.go
  • deploy/operator/api/v1beta1/dynamographdeployment_types.go
  • deploy/operator/api/v1beta1/zz_generated.deepcopy.go
  • deploy/operator/internal/controller/dynamographdeployment_controller.go
  • deploy/operator/internal/dynamo/grove.go
  • deploy/operator/internal/dynamo/grove_test.go

Comment on lines +292 to +300
if r.isGrovePathway(dynamoDeployment) {
pcs, pcsErr := r.getExistingGrovePodCliqueSet(ctx, dynamoDeployment)
if pcsErr != nil {
logger.V(1).Info("Failed to get PodCliqueSet for placement score", "error", pcsErr)
} else {
score, state := dynamo.AggregatePlacementScore(pcs)
dynamoDeployment.Status.PlacementScore = score
dynamoDeployment.Status.PlacementScoreState = state
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win

Reset placement status on non-Grove and lookup-error paths.

Lines 292-300 only overwrite these fields on the happy Grove path, so a DGD can keep stale status.placementScore* values after Grove is disabled or after a transient getExistingGrovePodCliqueSet failure. Please clear or explicitly set them on every reconcile path instead of leaving the previous status intact.

Suggested fix
-	if r.isGrovePathway(dynamoDeployment) {
+	dynamoDeployment.Status.PlacementScore = nil
+	dynamoDeployment.Status.PlacementScoreState = nvidiacomv1beta1.PlacementScoreStateUnsupported
+	if r.isGrovePathway(dynamoDeployment) {
 		pcs, pcsErr := r.getExistingGrovePodCliqueSet(ctx, dynamoDeployment)
 		if pcsErr != nil {
 			logger.V(1).Info("Failed to get PodCliqueSet for placement score", "error", pcsErr)
+			dynamoDeployment.Status.PlacementScoreState = nvidiacomv1beta1.PlacementScoreStateUnknown
 		} else {
 			score, state := dynamo.AggregatePlacementScore(pcs)
 			dynamoDeployment.Status.PlacementScore = score
 			dynamoDeployment.Status.PlacementScoreState = state
 		}
 	}
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
if r.isGrovePathway(dynamoDeployment) {
pcs, pcsErr := r.getExistingGrovePodCliqueSet(ctx, dynamoDeployment)
if pcsErr != nil {
logger.V(1).Info("Failed to get PodCliqueSet for placement score", "error", pcsErr)
} else {
score, state := dynamo.AggregatePlacementScore(pcs)
dynamoDeployment.Status.PlacementScore = score
dynamoDeployment.Status.PlacementScoreState = state
}
dynamoDeployment.Status.PlacementScore = nil
dynamoDeployment.Status.PlacementScoreState = nvidiacomv1beta1.PlacementScoreStateUnsupported
if r.isGrovePathway(dynamoDeployment) {
pcs, pcsErr := r.getExistingGrovePodCliqueSet(ctx, dynamoDeployment)
if pcsErr != nil {
logger.V(1).Info("Failed to get PodCliqueSet for placement score", "error", pcsErr)
dynamoDeployment.Status.PlacementScoreState = nvidiacomv1beta1.PlacementScoreStateUnknown
} else {
score, state := dynamo.AggregatePlacementScore(pcs)
dynamoDeployment.Status.PlacementScore = score
dynamoDeployment.Status.PlacementScoreState = state
}
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@deploy/operator/internal/controller/dynamographdeployment_controller.go`
around lines 292 - 300, Reset the placement status fields in
dynamographdeployment_controller.go so stale values do not survive non-Grove or
lookup-failure reconciles. In the reconcile logic around r.isGrovePathway and
getExistingGrovePodCliqueSet, explicitly clear or set
dynamoDeployment.Status.PlacementScore and
dynamoDeployment.Status.PlacementScoreState on the non-Grove branch and when
pcsErr is returned, instead of only updating them after AggregatePlacementScore
succeeds. Use the existing symbols r.isGrovePathway,
getExistingGrovePodCliqueSet, and dynamo.AggregatePlacementScore to place the
fix in the same status-update path.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

Open in Devin Review

Signed-off-by: ashnamehrotra <ashnamehrotra@gmail.com>
@ashnamehrotra
ashnamehrotra temporarily deployed to external_collaborator June 30, 2026 13:40 — with GitHub Actions Inactive
@dynamo-ops

Copy link
Copy Markdown
Contributor

/ok to test 0c56c9e

Signed-off-by: ashnamehrotra <ashnamehrotra@gmail.com>
@ashnamehrotra
ashnamehrotra requested a review from a team as a code owner June 30, 2026 16:33
@ashnamehrotra
ashnamehrotra temporarily deployed to external_collaborator June 30, 2026 16:33 — with GitHub Actions Inactive
@dynamo-ops

Copy link
Copy Markdown
Contributor

/ok to test de57d0f

@github-actions github-actions Bot added the documentation Improvements or additions to documentation label Jun 30, 2026
Comment thread deploy/operator/api/v1alpha1/dynamographdeployment_types.go Outdated
Comment thread deploy/operator/internal/controller/dynamographdeployment_controller.go Outdated
@ashnamehrotra
ashnamehrotra temporarily deployed to external_collaborator July 9, 2026 20:01 — with GitHub Actions Inactive
@dynamo-ops

Copy link
Copy Markdown
Contributor

/ok to test a93fe91

Resolve conflicts in the DGD controller (keep placement-status write,
drop the duplicate reconcile-error check that main removed) and in
grove.go (keep AggregatePlacementScore alongside main's shortened
comment on specToGroveTopologyConstraint). Regenerate deepcopy, CRDs,
api-reference, helm docs, and helm CRD copy to pick up both trees.

Signed-off-by: ashnamehrotra <ashnamehrotra@gmail.com>
@ashnamehrotra
ashnamehrotra temporarily deployed to external_collaborator July 9, 2026 20:10 — with GitHub Actions Inactive
@dynamo-ops

Copy link
Copy Markdown
Contributor

/ok to test e3b76e7

@ashnamehrotra
ashnamehrotra temporarily deployed to external_collaborator July 10, 2026 15:28 — with GitHub Actions Inactive
@dynamo-ops

Copy link
Copy Markdown
Contributor

/ok to test df4d992

Register no-op custom fuzzers for the v1alpha1 and v1beta1
PlacementStatus so the round-trip fuzz filler does not draw random
bits for Placement. Placement round-trip is already covered by
TestDGD_RoundTrip_Status.

Signed-off-by: ashnamehrotra <ashnamehrotra@gmail.com>
@ashnamehrotra
ashnamehrotra force-pushed the grove-placement-score branch from df4d992 to 5214cf8 Compare July 10, 2026 15:52
@ashnamehrotra
ashnamehrotra temporarily deployed to external_collaborator July 10, 2026 15:52 — with GitHub Actions Inactive
@dynamo-ops

Copy link
Copy Markdown
Contributor

/ok to test 5214cf8

Comment thread deploy/operator/internal/dynamo/grove.go Outdated
Comment thread deploy/operator/internal/controller/dynamographdeployment_controller.go Outdated
Comment thread deploy/operator/api/roundtrip_fuzz_test.go Outdated

sttts commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

Comparing this PR with DEP #10064, is the intent to land only the placement-status API/scaffolding here and implement the remaining DEP requirements in follow-ups?

In particular:

  • REQ 1: the Grove-specific Ready-condition diagnosis (insufficient_capacity, updating, pods_not_ready, and mixed causes)
  • REQ 2: portable component-level scheduledReplicas
  • REQ 3/4: the actual placement-score provider and Reported/Partial aggregation once the source question is resolved
  • REQ 7: the optional metric and low-score warning threshold, if those are still planned

If these are intentionally deferred, could we link the follow-up issues/PRs or make the partial DEP scope explicit in the PR description?

@devivasudevan

Copy link
Copy Markdown
Contributor

Comparing this PR with DEP #10064, is the intent to land only the placement-status API/scaffolding here and implement the remaining DEP requirements in follow-ups?

In particular:

  • REQ 1: the Grove-specific Ready-condition diagnosis (insufficient_capacity, updating, pods_not_ready, and mixed causes)
  • REQ 2: portable component-level scheduledReplicas
  • REQ 3/4: the actual placement-score provider and Reported/Partial aggregation once the source question is resolved
  • REQ 7: the optional metric and low-score warning threshold, if those are still planned

If these are intentionally deferred, could we link the follow-up issues/PRs or make the partial DEP scope explicit in the PR description?

@sttts : REQ 1 + REQ 2 is covered with this PR

Address @sttts P1 on grove.go:300. AggregatePlacementScore returned
Unsupported for every non-empty PodGang list, so Reported/Partial and
status.placement.score were unreachable until Grove exposes a placement
score. Until the DEP ai-dynamo#10064 open question on score source is resolved,
land the API shape (types, CRD, conversion, round-trip tests) without
wiring the controller.

Removed:
- dynamo.AggregatePlacementScore and its unit test
- controller.updatePlacementStatus and its call site in Reconcile

Kept:
- PlacementStatus struct with Score + State (v1beta1 + v1alpha1)
- Enum with Reported / Partial / Unsupported / Unknown
- Score CRD bounds [0, 1]
- v1alpha1<->v1beta1 conversion + round-trip test

Doc-comment on v1beta1 PlacementStatus notes the schema-only state and
points at DEP ai-dynamo#10064. Regenerated CRDs, api-reference, and helm CRD copy.

Signed-off-by: ashnamehrotra <ashnamehrotra@gmail.com>
@ashnamehrotra
ashnamehrotra requested review from a team as code owners July 16, 2026 14:33
@ashnamehrotra
ashnamehrotra temporarily deployed to external_collaborator July 16, 2026 14:33 — with GitHub Actions Inactive
@dynamo-ops

Copy link
Copy Markdown
Contributor

/ok to test 36d2d27

Address @sttts on roundtrip_fuzz_test.go:224. Replace the no-op
PlacementStatus fuzzers with real randomizers that pick State from the
four PlacementScoreState enum values and Score from either nil or a
value inside the CRD [0, 1] bounds. Add an oneOfPtr helper alongside
oneOf for the sometimes-nil-sometimes-value pattern that optional API
pointer fields need.

Signed-off-by: ashnamehrotra <ashnamehrotra@gmail.com>
@ashnamehrotra
ashnamehrotra temporarily deployed to external_collaborator July 16, 2026 15:05 — with GitHub Actions Inactive
@dynamo-ops

Copy link
Copy Markdown
Contributor

/ok to test effa9fa

Resolve modify/delete conflict on the operator helm-chart CRD copy
(upstream ai-dynamo#11689 stops bundling CRDs in the Helm subchart) by
accepting the upstream delete. Regenerate deepcopy, CRDs, api-reference,
and helm docs on top of the merged tree.

Signed-off-by: ashnamehrotra <ashnamehrotra@gmail.com>
@ashnamehrotra
ashnamehrotra temporarily deployed to external_collaborator July 16, 2026 15:46 — with GitHub Actions Inactive
@dynamo-ops

Copy link
Copy Markdown
Contributor

/ok to test f7eb656

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

deployment::k8s Relates to dynamo deployment in kubernetes documentation Improvements or additions to documentation external-contribution Pull request is from an external contributor feat size/L

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants